{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## First What is BERT?\n",
        "\n",
        "BERT stands for Bidirectional Encoder Representations from Transformers. The name itself gives us several clues to what BERT is all about.\n",
        "\n",
        "BERT architecture consists of several Transformer encoders stacked together. Each Transformer encoder encapsulates two sub-layers: a self-attention layer and a feed-forward layer.\n",
        "\n",
        "### There are two different BERT models:\n",
        "\n",
        "- BERT base, which is a BERT model consists of 12 layers of Transformer encoder, 12 attention heads, 768 hidden size, and 110M parameters.\n",
        "\n",
        "- BERT large, which is a BERT model consists of 24 layers of Transformer encoder,16 attention heads, 1024 hidden size, and 340 parameters.\n",
        "\n",
        "\n",
        "\n",
        "BERT Input and Output\n",
        "BERT model expects a sequence of tokens (words) as an input. In each sequence of tokens, there are two special tokens that BERT would expect as an input:\n",
        "\n",
        "- [CLS]: This is the first token of every sequence, which stands for classification token.\n",
        "- [SEP]: This is the token that makes BERT know which token belongs to which sequence. This special token is mainly important for a next sentence prediction task or question-answering task. If we only have one sequence, then this token will be appended to the end of the sequence.\n",
        "\n",
        "\n",
        "It is also important to note that the maximum size of tokens that can be fed into BERT model is 512. If the tokens in a sequence are less than 512, we can use padding to fill the unused token slots with [PAD] token. If the tokens in a sequence are longer than 512, then we need to do a truncation.\n",
        "\n",
        "And that’s all that BERT expects as input.\n",
        "\n",
        "BERT model then will output an embedding vector of size 768 in each of the tokens. We can use these vectors as an input for different kinds of NLP applications, whether it is text classification, next sentence prediction, Named-Entity-Recognition (NER), or question-answering.\n",
        "\n",
        "\n",
        "------------\n",
        "\n",
        "**For a text classification task**, we focus our attention on the embedding vector output from the special [CLS] token. This means that we’re going to use the embedding vector of size 768 from [CLS] token as an input for our classifier, which then will output a vector of size the number of classes in our classification task.\n",
        "\n",
        "-----------------------\n",
        "\n",
        "![Imgur](https://imgur.com/NpeB9vb.png)\n",
        "\n",
        "-------------------------"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fOaCES6J_FKK"
      },
      "source": [
        "## Extracting embeddings from pre-trained BERT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "o7PQZSHt_FKT"
      },
      "outputs": [],
      "source": [
        "!pip install transformers -q"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "qmWTWzRX_FKe"
      },
      "outputs": [],
      "source": [
        "from transformers import BertModel, BertTokenizer\n",
        "import torch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "swqqjz08_FKj"
      },
      "outputs": [],
      "source": [
        "model = BertModel.from_pretrained('bert-base-uncased')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OHqHZVvvSX8W"
      },
      "outputs": [],
      "source": [
        "sentence = 'She is a MachineLearning Engineer and works in California'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Understanding Token IDs\n",
        "\n",
        "The token ids are indices in a vocabulary.\n",
        "\n",
        "The ids themselves are not used during the training of a network, rather the ids are transformed into vectors.\n",
        "\n",
        "Say you are inputting three words, and their ids are 12,14, and 4. What is actually is given as input is three vectors (say each of n-dimension) where each id is mapped to a unique vector. These vectors could be one-hot, i.e 1 at the index 4 for the token Id 4 and rest zeros, or they could be pre-trained embedding like GloVe.\n",
        "\n",
        "-----------------\n",
        "\n",
        "![](2022-09-27-21-01-25.png)\n",
        "\n",
        "![](2022-09-27-21-02-47.png)\n",
        "\n",
        "\n",
        "The token ID specifically is used in the embedding layer, which you can see as a matrix where row indices are the token IDs.\n",
        "\n",
        "The token ID is the row ID in the embedding matrix. So every row is a token representation\n",
        "\n",
        "So one row for each item in the total vocabulary, for instance 30K rows for 30k tokens. \n",
        "\n",
        "\n",
        "Every token therefore has a (learned!) representation. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lPHa9dI5_FKn"
      },
      "outputs": [],
      "source": [
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FVn3-2XM_FKv"
      },
      "source": [
        "Tokenize the sentence and obtain the tokens:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "collapsed": true,
        "id": "4Ui-3oWr_FKw"
      },
      "outputs": [],
      "source": [
        "tokens = tokenizer.tokenize(sentence)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ssD9jeyZ0Rs4",
        "outputId": "41469617-e424-484b-91df-906b4ddd459b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['she',\n",
              " 'is',\n",
              " 'a',\n",
              " 'machine',\n",
              " '##lea',\n",
              " '##rn',\n",
              " '##ing',\n",
              " 'engineer',\n",
              " 'and',\n",
              " 'works',\n",
              " 'in',\n",
              " 'california']"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokens"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RSYRLcvV_FKy"
      },
      "source": [
        "Let's print the tokens:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rL8VgpcM_FK0"
      },
      "source": [
        "Now, we will add the [CLS] token at the beginning and [SEP] token at the end of the tokens list: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "collapsed": true,
        "id": "3CpARvDc_FK0"
      },
      "outputs": [],
      "source": [
        "tokens = ['[CLS]'] + tokens + ['[SEP]']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jbFEyrk7_FK2"
      },
      "source": [
        "Let's look at our updated tokens list:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Uaf_JScw_FK3",
        "outputId": "10fe8fe9-dc71-4aea-af92-3ff66fb10452"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['[CLS]', 'she', 'is', 'a', 'machine', '##lea', '##rn', '##ing', 'engineer', 'and', 'works', 'in', 'california', '[SEP]']\n"
          ]
        }
      ],
      "source": [
        "print(tokens)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UZkIvB8E11Sj",
        "outputId": "80bb4899-eb99-439d-ef6b-d9ca0fc15420"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "14"
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(tokens)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "collapsed": true,
        "id": "cqMI--rA_FK4"
      },
      "outputs": [],
      "source": [
        "tokens = tokens + ['[PAD]'] + ['[PAD]']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XEEHs7CV_FK6"
      },
      "source": [
        "Let's print our updated tokens list:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2tKmh_hk_FK6",
        "outputId": "5fd5bd92-0779-4674-af46-8bc0f232cf18"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "16\n"
          ]
        }
      ],
      "source": [
        "print(len(tokens))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "collapsed": true,
        "id": "3SbiaBws_FK8"
      },
      "outputs": [],
      "source": [
        "attention_mask = [1 if i!= '[PAD]' else 0 for i in tokens]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NmtZKFFq_FK-",
        "outputId": "1c1df59c-2f47-4f0f-f112-6421755f135c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]\n"
          ]
        }
      ],
      "source": [
        "print(attention_mask)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VMtepIn13BXt"
      },
      "source": [
        "# unique token ID"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "collapsed": true,
        "id": "KGbsHfxL_FK_"
      },
      "outputs": [],
      "source": [
        "token_ids = tokenizer.convert_tokens_to_ids(tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zaQWwz_y_FK_"
      },
      "source": [
        "\n",
        "Let's have a look at the token_ids:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iih_wWeT_FLA",
        "outputId": "3b12a664-2ba7-4433-e71a-f71189d05b64"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[101, 2016, 2003, 1037, 3698, 19738, 6826, 2075, 3992, 1998, 2573, 1999, 2662, 102, 0, 0]\n"
          ]
        }
      ],
      "source": [
        "print(token_ids)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JQEtl8aF3Rov"
      },
      "outputs": [],
      "source": [
        "['[CLS]', 'she', 'is', 'a', 'machine', '##lea', '##rn', '##ing', 'engineer', 'and', 'works', 'in', 'california', '[SEP]']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "WxZXk6ZJ3Rbn"
      },
      "outputs": [],
      "source": [
        "token_ids = torch.tensor(token_ids).unsqueeze(0)\n",
        "\n",
        "attention_mask = torch.tensor(attention_mask).unsqueeze(0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pQn5de_a_FLC"
      },
      "source": [
        "\n",
        "That's it. Next, we feed the token_ids and attention_mask to the pre-trained BERT model and get the embedding. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qYIDWAO-_FLD"
      },
      "source": [
        "## Getting the embedding \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "collapsed": true,
        "id": "Rh-Ohh71_FLE"
      },
      "outputs": [],
      "source": [
        "output = model(token_ids, attention_mask = attention_mask)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-Cm5HOWWSX8f",
        "outputId": "c8ccf480-d7c0-4b26-9e45-5493053a6986"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "BaseModelOutputWithPoolingAndCrossAttentions(last_hidden_state=tensor([[[-0.1925,  0.1684, -0.4252,  ..., -0.2599,  0.3736,  0.0529],\n",
              "         [ 0.2417, -0.2748, -0.4909,  ...,  0.1372,  0.3408, -0.4655],\n",
              "         [-0.0871,  0.0837,  0.2605,  ..., -0.4635, -0.0462,  0.2621],\n",
              "         ...,\n",
              "         [ 0.6711, -0.0076, -0.3847,  ..., -0.1289, -0.5171, -0.8002],\n",
              "         [-0.2731,  0.1098, -0.5440,  ...,  0.0314,  0.4467, -0.3448],\n",
              "         [-0.2387,  0.0119, -0.4760,  ...,  0.4656,  0.5837, -0.3774]]],\n",
              "       grad_fn=<NativeLayerNormBackward0>), pooler_output=tensor([[-0.9531, -0.4914, -0.8872,  0.9035,  0.8174, -0.2919,  0.9511,  0.4982,\n",
              "         -0.7595, -1.0000, -0.6996,  0.9459,  0.9890,  0.4754,  0.9723, -0.8460,\n",
              "         -0.1423, -0.7209,  0.4428, -0.7905,  0.7822,  1.0000,  0.2119,  0.4066,\n",
              "          0.5813,  0.9923, -0.8380,  0.9670,  0.9746,  0.8324, -0.8227,  0.4136,\n",
              "         -0.9931, -0.2821, -0.8860, -0.9961,  0.5261, -0.8722, -0.0915, -0.0950,\n",
              "         -0.9237,  0.5106,  1.0000, -0.0830,  0.5382, -0.3140, -1.0000,  0.3774,\n",
              "         -0.9557,  0.8998,  0.7947,  0.8279,  0.2756,  0.6581,  0.6064, -0.3369,\n",
              "          0.0251,  0.1856, -0.3297, -0.7515, -0.6843,  0.4392, -0.8613, -0.9603,\n",
              "          0.8838,  0.7763, -0.3041, -0.3105, -0.1854, -0.0969,  0.9726,  0.3039,\n",
              "          0.0437, -0.8890,  0.6619,  0.2710, -0.7410,  1.0000, -0.5422, -0.9900,\n",
              "          0.7010,  0.7629,  0.6910, -0.1635,  0.4133, -1.0000,  0.6527, -0.1595,\n",
              "         -0.9959,  0.2069,  0.5956, -0.3285,  0.3339,  0.7146, -0.4482, -0.5843,\n",
              "         -0.4799, -0.8426, -0.3858, -0.5225,  0.1742, -0.3364, -0.4200, -0.5325,\n",
              "          0.2888, -0.6126, -0.6725,  0.4133, -0.0447,  0.7262,  0.5707, -0.4401,\n",
              "          0.4878, -0.9724,  0.7432, -0.3434, -0.9916, -0.7343, -0.9950,  0.8367,\n",
              "         -0.4385, -0.3597,  0.9864,  0.2272,  0.5367, -0.2114, -0.9175, -1.0000,\n",
              "         -0.6876, -0.2065, -0.1163, -0.2588, -0.9895, -0.9715,  0.6886,  0.9719,\n",
              "          0.3643,  1.0000, -0.5358,  0.9622, -0.3474, -0.7841,  0.5620, -0.5971,\n",
              "          0.7511,  0.5313, -0.8059,  0.3467, -0.6067,  0.2069, -0.7716, -0.3528,\n",
              "         -0.5622, -0.9675, -0.4585,  0.9715, -0.6763, -0.9285,  0.0471, -0.4429,\n",
              "         -0.5263,  0.9179,  0.7994,  0.4794, -0.3966,  0.5520,  0.2816,  0.6815,\n",
              "         -0.8935, -0.2189,  0.5414, -0.4752, -0.7903, -0.9888, -0.5437,  0.7355,\n",
              "          0.9930,  0.8657,  0.3857,  0.8296, -0.4537,  0.8017, -0.9742,  0.9907,\n",
              "         -0.2973,  0.2552, -0.0693,  0.2771, -0.9199, -0.0733,  0.9234, -0.8223,\n",
              "         -0.9357, -0.1388, -0.5252, -0.5398, -0.7973,  0.6874, -0.4210, -0.5393,\n",
              "         -0.2636,  0.9614,  0.9941,  0.8501,  0.1416,  0.7740, -0.9695, -0.5676,\n",
              "          0.2402,  0.4135,  0.2547,  0.9958, -0.6463, -0.3107, -0.9529, -0.9896,\n",
              "          0.0218, -0.9544, -0.2193, -0.7695,  0.7273, -0.6121,  0.5897,  0.5309,\n",
              "         -0.9961, -0.8619,  0.4588, -0.4799,  0.5743, -0.2629,  0.5680,  0.9034,\n",
              "         -0.7476,  0.7253,  0.9435, -0.8031, -0.8828,  0.9233, -0.3802,  0.9358,\n",
              "         -0.7946,  0.9985,  0.9041,  0.8270, -0.9735, -0.6009, -0.9616, -0.6920,\n",
              "         -0.1396,  0.0737,  0.8640,  0.7403,  0.5014,  0.5310, -0.6402,  0.9996,\n",
              "         -0.7336, -0.9676, -0.2554, -0.3897, -0.9930,  0.8208,  0.3768,  0.3227,\n",
              "         -0.5386, -0.7843, -0.9719,  0.9473,  0.3165,  0.9974, -0.4319, -0.9673,\n",
              "         -0.7071, -0.9575, -0.0163, -0.2454, -0.2759, -0.0372, -0.9812,  0.5801,\n",
              "          0.6953,  0.7258, -0.8238,  0.9997,  1.0000,  0.9829,  0.9496,  0.9723,\n",
              "         -0.9999, -0.6599,  1.0000, -0.9929, -1.0000, -0.9784, -0.7252,  0.4299,\n",
              "         -1.0000, -0.1801, -0.0305, -0.9492,  0.6348,  0.9822,  0.9986, -1.0000,\n",
              "          0.9494,  0.9772, -0.7665,  0.9494, -0.4603,  0.9812,  0.4982,  0.6965,\n",
              "         -0.4254,  0.5564, -0.9028, -0.9304, -0.6293, -0.7504,  0.9983,  0.1893,\n",
              "         -0.8871, -0.9579,  0.6570, -0.0046, -0.2352, -0.9797, -0.3472,  0.5996,\n",
              "          0.7439,  0.3234,  0.4430, -0.8331,  0.3286, -0.2771,  0.6577,  0.7548,\n",
              "         -0.9564, -0.7847, -0.2250, -0.1100, -0.5044, -0.9741,  0.9876, -0.6114,\n",
              "          0.8556,  1.0000,  0.1821, -0.9664,  0.7199,  0.3572,  0.0222,  1.0000,\n",
              "          0.8395, -0.9918, -0.6857,  0.7819, -0.6987, -0.7588,  0.9999, -0.4085,\n",
              "         -0.6102, -0.4808,  0.9868, -0.9927,  0.9941, -0.9452, -0.9816,  0.9861,\n",
              "          0.9683, -0.5740, -0.8463,  0.2055, -0.6551,  0.2891, -0.9839,  0.8015,\n",
              "          0.5615, -0.2211,  0.9360, -0.8772, -0.6209,  0.4443, -0.4753, -0.0369,\n",
              "          0.9119,  0.5990, -0.2902,  0.1669, -0.4288, -0.1507, -0.9872,  0.5583,\n",
              "          1.0000, -0.1425,  0.6010, -0.4927, -0.2014,  0.0381,  0.6139,  0.6655,\n",
              "         -0.3497, -0.9447,  0.6909, -0.9862, -0.9938,  0.8442,  0.2112, -0.3318,\n",
              "          1.0000,  0.5579,  0.3544,  0.3334,  0.9877, -0.0054,  0.7005,  0.8336,\n",
              "          0.9878, -0.3566,  0.6850,  0.9445, -0.8846, -0.4321, -0.7359,  0.0560,\n",
              "         -0.9299,  0.0186, -0.9857,  0.9844,  0.9458,  0.4526,  0.3714,  0.6404,\n",
              "          1.0000, -0.6144,  0.7733, -0.5719,  0.8266, -0.9998, -0.9343, -0.4705,\n",
              "         -0.0299, -0.7185, -0.3312,  0.3188, -0.9867,  0.7307,  0.7287, -0.9946,\n",
              "         -0.9925, -0.1620,  0.9231,  0.2522, -0.9828, -0.8127, -0.6838,  0.8036,\n",
              "         -0.3376, -0.9687, -0.0425, -0.4425,  0.6618, -0.3970,  0.6552,  0.8037,\n",
              "          0.7465, -0.6848, -0.5180, -0.2658, -0.8857,  0.8209, -0.9260, -0.9383,\n",
              "         -0.2320,  1.0000, -0.3935,  0.7685,  0.7838,  0.8586, -0.2762,  0.2479,\n",
              "          0.9251,  0.3105, -0.6583, -0.8387, -0.7902, -0.4746,  0.6613,  0.3836,\n",
              "          0.5403,  0.8679,  0.8154,  0.4072, -0.0469,  0.1889,  0.9999, -0.1608,\n",
              "         -0.3208, -0.8006, -0.2484, -0.4890, -0.3771,  1.0000,  0.4563,  0.6000,\n",
              "         -0.9953, -0.8646, -0.9801,  1.0000,  0.8901, -0.9555,  0.7847,  0.6954,\n",
              "         -0.2841,  0.8547, -0.4522, -0.4141,  0.2546,  0.1530,  0.9803, -0.6607,\n",
              "         -0.9827, -0.6989,  0.6511, -0.9846,  1.0000, -0.7535, -0.3579, -0.4932,\n",
              "         -0.4461,  0.6383, -0.0279, -0.9917, -0.3726,  0.2336,  0.9893,  0.3576,\n",
              "         -0.6986, -0.9609,  0.8582,  0.7506, -0.8731, -0.9709,  0.9808, -0.9894,\n",
              "          0.5911,  1.0000,  0.4033,  0.1263,  0.3721, -0.5706,  0.4401, -0.5306,\n",
              "          0.7562, -0.9794, -0.4814, -0.2390,  0.5132, -0.2051, -0.3140,  0.7925,\n",
              "          0.2790, -0.6370, -0.7084, -0.2946,  0.5667,  0.8821, -0.3569, -0.1902,\n",
              "          0.2061, -0.1334, -0.9783, -0.5647, -0.5330, -1.0000,  0.8590, -1.0000,\n",
              "          0.5967,  0.1381, -0.2933,  0.9095,  0.6262,  0.6778, -0.8746, -0.7993,\n",
              "          0.6081,  0.8491, -0.5739, -0.5696, -0.8015,  0.4588, -0.2228,  0.4722,\n",
              "         -0.5556,  0.7975, -0.4495,  1.0000,  0.2676, -0.6018, -0.9924,  0.3877,\n",
              "         -0.3227,  1.0000, -0.9546, -0.9756,  0.4031, -0.8655, -0.9055,  0.4154,\n",
              "          0.0666, -0.8774, -0.9569,  0.9840,  0.9531, -0.6600,  0.5991, -0.3792,\n",
              "         -0.6686,  0.0022,  0.9049,  0.9929,  0.3449,  0.9516, -0.2504, -0.0766,\n",
              "          0.9860,  0.2324,  0.7526,  0.2225,  1.0000,  0.4340, -0.9459,  0.1673,\n",
              "         -0.9949, -0.3226, -0.9685,  0.4636,  0.2128,  0.9480, -0.4177,  0.9806,\n",
              "         -0.8839,  0.1223, -0.6722, -0.4115,  0.4124, -0.9792, -0.9900, -0.9896,\n",
              "          0.7181, -0.5447, -0.2199,  0.3385,  0.1928,  0.6341,  0.6016, -1.0000,\n",
              "          0.9638,  0.5932,  0.9012,  0.9819,  0.6336,  0.6025,  0.4647, -0.9921,\n",
              "         -0.9960, -0.4777, -0.3020,  0.8082,  0.8069,  0.8981,  0.4994, -0.5467,\n",
              "         -0.5973, -0.3117, -0.7843, -0.9962,  0.6334, -0.6465, -0.9882,  0.9734,\n",
              "         -0.2728, -0.1237,  0.1477, -0.6318,  0.9864,  0.9219,  0.5294,  0.2008,\n",
              "          0.6393,  0.9530,  0.9745,  0.9906, -0.8397,  0.9371, -0.8450,  0.4911,\n",
              "          0.5586, -0.9563,  0.2780,  0.6846, -0.4208,  0.3979, -0.2707, -0.9895,\n",
              "          0.6312, -0.3565,  0.6168, -0.5852, -0.0086, -0.5205, -0.2230, -0.8108,\n",
              "         -0.7440,  0.7475,  0.6836,  0.9567,  0.8605, -0.2461, -0.8531, -0.2549,\n",
              "         -0.6971, -0.9461,  0.9677, -0.1343, -0.2094,  0.6902, -0.0374,  0.8592,\n",
              "          0.0442, -0.5071, -0.5022, -0.8493,  0.9526, -0.6960, -0.6532, -0.5678,\n",
              "          0.8615,  0.4444,  1.0000, -0.7972, -0.8532, -0.6103, -0.5343,  0.4503,\n",
              "         -0.5558, -1.0000,  0.4520, -0.6910,  0.5643, -0.6321,  0.8368, -0.7881,\n",
              "         -0.9914, -0.3622,  0.4591,  0.7768, -0.5997, -0.8041,  0.7214, -0.2333,\n",
              "          0.9777,  0.9188, -0.6901,  0.1259,  0.7553, -0.4155, -0.6854,  0.9550]],\n",
              "       grad_fn=<TanhBackward0>), hidden_states=None, past_key_values=None, attentions=None, cross_attentions=None)"
            ]
          },
          "execution_count": 41,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gx1RfsaU_FLF",
        "outputId": "9db02b7d-6b1b-4651-847e-81d7143b71e6"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 768])"
            ]
          },
          "execution_count": 44,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "output[0].shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zAL6KEXw_FLI",
        "outputId": "61a46f62-cea2-474d-ba3e-77037f2364b5"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "tensor([[-0.9531, -0.4914, -0.8872,  0.9035,  0.8174, -0.2919,  0.9511,  0.4982,\n",
              "         -0.7595, -1.0000, -0.6996,  0.9459,  0.9890,  0.4754,  0.9723, -0.8460,\n",
              "         -0.1423, -0.7209,  0.4428, -0.7905,  0.7822,  1.0000,  0.2119,  0.4066,\n",
              "          0.5813,  0.9923, -0.8380,  0.9670,  0.9746,  0.8324, -0.8227,  0.4136,\n",
              "         -0.9931, -0.2821, -0.8860, -0.9961,  0.5261, -0.8722, -0.0915, -0.0950,\n",
              "         -0.9237,  0.5106,  1.0000, -0.0830,  0.5382, -0.3140, -1.0000,  0.3774,\n",
              "         -0.9557,  0.8998,  0.7947,  0.8279,  0.2756,  0.6581,  0.6064, -0.3369,\n",
              "          0.0251,  0.1856, -0.3297, -0.7515, -0.6843,  0.4392, -0.8613, -0.9603,\n",
              "          0.8838,  0.7763, -0.3041, -0.3105, -0.1854, -0.0969,  0.9726,  0.3039,\n",
              "          0.0437, -0.8890,  0.6619,  0.2710, -0.7410,  1.0000, -0.5422, -0.9900,\n",
              "          0.7010,  0.7629,  0.6910, -0.1635,  0.4133, -1.0000,  0.6527, -0.1595,\n",
              "         -0.9959,  0.2069,  0.5956, -0.3285,  0.3339,  0.7146, -0.4482, -0.5843,\n",
              "         -0.4799, -0.8426, -0.3858, -0.5225,  0.1742, -0.3364, -0.4200, -0.5325,\n",
              "          0.2888, -0.6126, -0.6725,  0.4133, -0.0447,  0.7262,  0.5707, -0.4401,\n",
              "          0.4878, -0.9724,  0.7432, -0.3434, -0.9916, -0.7343, -0.9950,  0.8367,\n",
              "         -0.4385, -0.3597,  0.9864,  0.2272,  0.5367, -0.2114, -0.9175, -1.0000,\n",
              "         -0.6876, -0.2065, -0.1163, -0.2588, -0.9895, -0.9715,  0.6886,  0.9719,\n",
              "          0.3643,  1.0000, -0.5358,  0.9622, -0.3474, -0.7841,  0.5620, -0.5971,\n",
              "          0.7511,  0.5313, -0.8059,  0.3467, -0.6067,  0.2069, -0.7716, -0.3528,\n",
              "         -0.5622, -0.9675, -0.4585,  0.9715, -0.6763, -0.9285,  0.0471, -0.4429,\n",
              "         -0.5263,  0.9179,  0.7994,  0.4794, -0.3966,  0.5520,  0.2816,  0.6815,\n",
              "         -0.8935, -0.2189,  0.5414, -0.4752, -0.7903, -0.9888, -0.5437,  0.7355,\n",
              "          0.9930,  0.8657,  0.3857,  0.8296, -0.4537,  0.8017, -0.9742,  0.9907,\n",
              "         -0.2973,  0.2552, -0.0693,  0.2771, -0.9199, -0.0733,  0.9234, -0.8223,\n",
              "         -0.9357, -0.1388, -0.5252, -0.5398, -0.7973,  0.6874, -0.4210, -0.5393,\n",
              "         -0.2636,  0.9614,  0.9941,  0.8501,  0.1416,  0.7740, -0.9695, -0.5676,\n",
              "          0.2402,  0.4135,  0.2547,  0.9958, -0.6463, -0.3107, -0.9529, -0.9896,\n",
              "          0.0218, -0.9544, -0.2193, -0.7695,  0.7273, -0.6121,  0.5897,  0.5309,\n",
              "         -0.9961, -0.8619,  0.4588, -0.4799,  0.5743, -0.2629,  0.5680,  0.9034,\n",
              "         -0.7476,  0.7253,  0.9435, -0.8031, -0.8828,  0.9233, -0.3802,  0.9358,\n",
              "         -0.7946,  0.9985,  0.9041,  0.8270, -0.9735, -0.6009, -0.9616, -0.6920,\n",
              "         -0.1396,  0.0737,  0.8640,  0.7403,  0.5014,  0.5310, -0.6402,  0.9996,\n",
              "         -0.7336, -0.9676, -0.2554, -0.3897, -0.9930,  0.8208,  0.3768,  0.3227,\n",
              "         -0.5386, -0.7843, -0.9719,  0.9473,  0.3165,  0.9974, -0.4319, -0.9673,\n",
              "         -0.7071, -0.9575, -0.0163, -0.2454, -0.2759, -0.0372, -0.9812,  0.5801,\n",
              "          0.6953,  0.7258, -0.8238,  0.9997,  1.0000,  0.9829,  0.9496,  0.9723,\n",
              "         -0.9999, -0.6599,  1.0000, -0.9929, -1.0000, -0.9784, -0.7252,  0.4299,\n",
              "         -1.0000, -0.1801, -0.0305, -0.9492,  0.6348,  0.9822,  0.9986, -1.0000,\n",
              "          0.9494,  0.9772, -0.7665,  0.9494, -0.4603,  0.9812,  0.4982,  0.6965,\n",
              "         -0.4254,  0.5564, -0.9028, -0.9304, -0.6293, -0.7504,  0.9983,  0.1893,\n",
              "         -0.8871, -0.9579,  0.6570, -0.0046, -0.2352, -0.9797, -0.3472,  0.5996,\n",
              "          0.7439,  0.3234,  0.4430, -0.8331,  0.3286, -0.2771,  0.6577,  0.7548,\n",
              "         -0.9564, -0.7847, -0.2250, -0.1100, -0.5044, -0.9741,  0.9876, -0.6114,\n",
              "          0.8556,  1.0000,  0.1821, -0.9664,  0.7199,  0.3572,  0.0222,  1.0000,\n",
              "          0.8395, -0.9918, -0.6857,  0.7819, -0.6987, -0.7588,  0.9999, -0.4085,\n",
              "         -0.6102, -0.4808,  0.9868, -0.9927,  0.9941, -0.9452, -0.9816,  0.9861,\n",
              "          0.9683, -0.5740, -0.8463,  0.2055, -0.6551,  0.2891, -0.9839,  0.8015,\n",
              "          0.5615, -0.2211,  0.9360, -0.8772, -0.6209,  0.4443, -0.4753, -0.0369,\n",
              "          0.9119,  0.5990, -0.2902,  0.1669, -0.4288, -0.1507, -0.9872,  0.5583,\n",
              "          1.0000, -0.1425,  0.6010, -0.4927, -0.2014,  0.0381,  0.6139,  0.6655,\n",
              "         -0.3497, -0.9447,  0.6909, -0.9862, -0.9938,  0.8442,  0.2112, -0.3318,\n",
              "          1.0000,  0.5579,  0.3544,  0.3334,  0.9877, -0.0054,  0.7005,  0.8336,\n",
              "          0.9878, -0.3566,  0.6850,  0.9445, -0.8846, -0.4321, -0.7359,  0.0560,\n",
              "         -0.9299,  0.0186, -0.9857,  0.9844,  0.9458,  0.4526,  0.3714,  0.6404,\n",
              "          1.0000, -0.6144,  0.7733, -0.5719,  0.8266, -0.9998, -0.9343, -0.4705,\n",
              "         -0.0299, -0.7185, -0.3312,  0.3188, -0.9867,  0.7307,  0.7287, -0.9946,\n",
              "         -0.9925, -0.1620,  0.9231,  0.2522, -0.9828, -0.8127, -0.6838,  0.8036,\n",
              "         -0.3376, -0.9687, -0.0425, -0.4425,  0.6618, -0.3970,  0.6552,  0.8037,\n",
              "          0.7465, -0.6848, -0.5180, -0.2658, -0.8857,  0.8209, -0.9260, -0.9383,\n",
              "         -0.2320,  1.0000, -0.3935,  0.7685,  0.7838,  0.8586, -0.2762,  0.2479,\n",
              "          0.9251,  0.3105, -0.6583, -0.8387, -0.7902, -0.4746,  0.6613,  0.3836,\n",
              "          0.5403,  0.8679,  0.8154,  0.4072, -0.0469,  0.1889,  0.9999, -0.1608,\n",
              "         -0.3208, -0.8006, -0.2484, -0.4890, -0.3771,  1.0000,  0.4563,  0.6000,\n",
              "         -0.9953, -0.8646, -0.9801,  1.0000,  0.8901, -0.9555,  0.7847,  0.6954,\n",
              "         -0.2841,  0.8547, -0.4522, -0.4141,  0.2546,  0.1530,  0.9803, -0.6607,\n",
              "         -0.9827, -0.6989,  0.6511, -0.9846,  1.0000, -0.7535, -0.3579, -0.4932,\n",
              "         -0.4461,  0.6383, -0.0279, -0.9917, -0.3726,  0.2336,  0.9893,  0.3576,\n",
              "         -0.6986, -0.9609,  0.8582,  0.7506, -0.8731, -0.9709,  0.9808, -0.9894,\n",
              "          0.5911,  1.0000,  0.4033,  0.1263,  0.3721, -0.5706,  0.4401, -0.5306,\n",
              "          0.7562, -0.9794, -0.4814, -0.2390,  0.5132, -0.2051, -0.3140,  0.7925,\n",
              "          0.2790, -0.6370, -0.7084, -0.2946,  0.5667,  0.8821, -0.3569, -0.1902,\n",
              "          0.2061, -0.1334, -0.9783, -0.5647, -0.5330, -1.0000,  0.8590, -1.0000,\n",
              "          0.5967,  0.1381, -0.2933,  0.9095,  0.6262,  0.6778, -0.8746, -0.7993,\n",
              "          0.6081,  0.8491, -0.5739, -0.5696, -0.8015,  0.4588, -0.2228,  0.4722,\n",
              "         -0.5556,  0.7975, -0.4495,  1.0000,  0.2676, -0.6018, -0.9924,  0.3877,\n",
              "         -0.3227,  1.0000, -0.9546, -0.9756,  0.4031, -0.8655, -0.9055,  0.4154,\n",
              "          0.0666, -0.8774, -0.9569,  0.9840,  0.9531, -0.6600,  0.5991, -0.3792,\n",
              "         -0.6686,  0.0022,  0.9049,  0.9929,  0.3449,  0.9516, -0.2504, -0.0766,\n",
              "          0.9860,  0.2324,  0.7526,  0.2225,  1.0000,  0.4340, -0.9459,  0.1673,\n",
              "         -0.9949, -0.3226, -0.9685,  0.4636,  0.2128,  0.9480, -0.4177,  0.9806,\n",
              "         -0.8839,  0.1223, -0.6722, -0.4115,  0.4124, -0.9792, -0.9900, -0.9896,\n",
              "          0.7181, -0.5447, -0.2199,  0.3385,  0.1928,  0.6341,  0.6016, -1.0000,\n",
              "          0.9638,  0.5932,  0.9012,  0.9819,  0.6336,  0.6025,  0.4647, -0.9921,\n",
              "         -0.9960, -0.4777, -0.3020,  0.8082,  0.8069,  0.8981,  0.4994, -0.5467,\n",
              "         -0.5973, -0.3117, -0.7843, -0.9962,  0.6334, -0.6465, -0.9882,  0.9734,\n",
              "         -0.2728, -0.1237,  0.1477, -0.6318,  0.9864,  0.9219,  0.5294,  0.2008,\n",
              "          0.6393,  0.9530,  0.9745,  0.9906, -0.8397,  0.9371, -0.8450,  0.4911,\n",
              "          0.5586, -0.9563,  0.2780,  0.6846, -0.4208,  0.3979, -0.2707, -0.9895,\n",
              "          0.6312, -0.3565,  0.6168, -0.5852, -0.0086, -0.5205, -0.2230, -0.8108,\n",
              "         -0.7440,  0.7475,  0.6836,  0.9567,  0.8605, -0.2461, -0.8531, -0.2549,\n",
              "         -0.6971, -0.9461,  0.9677, -0.1343, -0.2094,  0.6902, -0.0374,  0.8592,\n",
              "          0.0442, -0.5071, -0.5022, -0.8493,  0.9526, -0.6960, -0.6532, -0.5678,\n",
              "          0.8615,  0.4444,  1.0000, -0.7972, -0.8532, -0.6103, -0.5343,  0.4503,\n",
              "         -0.5558, -1.0000,  0.4520, -0.6910,  0.5643, -0.6321,  0.8368, -0.7881,\n",
              "         -0.9914, -0.3622,  0.4591,  0.7768, -0.5997, -0.8041,  0.7214, -0.2333,\n",
              "          0.9777,  0.9188, -0.6901,  0.1259,  0.7553, -0.4155, -0.6854,  0.9550]],\n",
              "       grad_fn=<TanhBackward0>)"
            ]
          },
          "execution_count": 45,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "output[1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mc6yVb0dDGPO",
        "outputId": "9877318b-3d57-40e7-f2b4-2f648ff6ccc9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "torch.Size([1, 768])"
            ]
          },
          "execution_count": 46,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "output[1].shape"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.8.10 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "vscode": {
      "interpreter": {
        "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
